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KEY MESSAGE ... 

College-level learning can be assessed 
in a cost-effective and minimally intrusive 
way to give states credible, comprehensive 
and comparable information. States can use 
the results to improve their higher education 
policies and practices, in order to increase 
the knowledge and skills of their residents. 

QUICK LOOK ... 

★ The extent of college-level learning varies 
from state to state and, within each state, 
from group to group. 

★ The pattern of learning varies from 
state to state, leading to different 
policy challenges. 

★ The racial and ethnic groups that are 
growing the fastest lag the most in 
learning, and more so in some states 
than in others. 

★ Teachers are not equally well prepared 
across states. 

★ All states should be able to use 
the same assessment strategy to get 
similar information. 

NEXT STEPS ... 

★ Each state should adopt the model 
described here, as recommended by 
the SHEEO (State Higher Education 
Executive Officers) National Commission 
on Accountability in Higher Education. 

★ Once results are in, state officials should 
convene policy and higher education 
leaders to discuss the findings, identify 
challenges, and target state policies 
and resources that will help address 
those challenges. 

★ Institutions should use the results 
to benchmark their performance 
and enrich their campus-based 
assessment programs. 



ASSESSING COLLEGE-LEVEL 
LEARNING 

By Margaret A. Miller 



A recent five-state project, the National Forum on College- 
Level Learning, has demonstrated that learning, 
higher education's most important product, can be assessed 
in ways that make interstate comparison possible, that these 
assessments are consistent with other information we have about 
the states, and that the results can be useful to policymakers. 

In a knowledge-based global economy, the economic, social, 
and civic welfare of states depends on the capacity of their 
residents to think at an increasingly sophisticated level. 
Certificates and degrees are increasingly inadequate proxies 
for this kind of "educational capital." It is the skills and knowledge 
behind the degrees that matters. A state-level approach to 
assessing college-level learning can give states that essential 
information. 

A focus on the state as a whole can tell policymakers: 

★ the extent to which its institutions are collectively effective 
in contributing to its store of educational capital, 

★ how well higher education is serving various regions 
or sub-populations within the state, 

★ how well the state's workforce-development efforts 
are working, 

it whether higher education is producing enough well-trained 
professionals in areas critical to the state's welfare, 

★ what economic development options are available or lacking 
to the state because of the educational capital it has, and 

★ whether the state has the range of programs needed 
for the economy and lifestyles its residents want. 

A collective examination also enables cost-benefit analyses 
concerning the learning that the state's system of higher 
education is producing relative to the state's investment. 

Armed with answers to these kinds of questions, a state can 
undertake further analyses, target resources where they are most 
needed to address urgent state priorities, and promote collective 
solutions to collective problems. 




NATIONAL ACCOUNTABILITY 
MOVEMENT 

Since the development of the 1990 
National Education Goals, the federal 
government has encouraged higher 
education to demonstrate the capacities 
of college graduates to communicate, 
solve problems, and think critically. 

Due to a lack of common 
benchmarks to compare student learning 
outcomes across states. Measuring Up 
2000, the first state-by-state report card 
on higher education performance, gave 
all states an 'Incomplete' in the category 
of learning. 

Shortly after the release of Measuring 
Up, the National Center convened an 
invitational forum of public-policy, 
education, and business leaders to 
examine how student learning could be 
measured at the state level. Between 
2002 and 2004, the National Forum 
on College-Level Learning piloted the 
newly developed assessment model in 
five states, demonstrating that providing 
comparative state-by-state information is 
not only feasible, but also important 
and useful for policy. 

The 2001 No Child Left Behind 
(NCLB) Act mandated that K-12 
education demonstrate its commitment 
to standards and educational equity 
through evidence of learning. Now 
a similar demand is beginning to be felt 
in higher education, particularly since 
recently released results of the National 
Assessment of Adult Literacy survey 
have shown that certain measures 
of American college graduate literacy 
have decreased significantly over 
the last decade. 

Most recently, U.S. Secretary 
of Education Margaret Spellings' 
Commission on the Future of Higher 
Education has been discussing how 
higher education might be held 
accountable for its results. In a 2006 
memo to the commissioners, Chairman 
Charles Miller pointed to the National 
Forum's project as evidence that those 
results can be measured in a cost- 
effective, minimally intrusive way. 

For more information 
on the National Forum, visit 
http: / / collegelevellearning.org. 



CURRENT STATE ASSESSMENT PLANS 
INADEQUATE FOR POLICY PURPOSES 

Most states have some kind of statewide assessment requirement 
in place to improve performance and/or give state officials a sense 
of what their investment in higher education has yielded. 

But the information they have on statewide collegiate learning is 
incomplete. Even those states that employ common measures 
for public colleges and universities know virtually nothing 
about the learning results of their private institutions. 

And when every public campus within a state assesses its 
students' learning differently, the state has no way to interpret 
the resulting information, because there are no external benchmarks 
against which to measure a given program's or institution's 
performance. Nor does the state know how the learning of its college- 
educated residents or current college attendees compares to the 
learning of those in other states — hence what its competitive position 
is with regard to its educational capital. 

Comparable assessment allows a state to: 

★ chart its progress in developing its educational capital, 

★ compare its performance to those of like states, and 

★ identify good practices. 

Given sample sizes that are large and sufficiently representative, 
institutions too can see how well they perform relative to their peers 
on key assessment measures. These external benchmarks can serve 
to anchor their more extensive campus-based assessment methods, 
which continue to be essential to improvement. 



THE LEARNING PROFILES 

The states in the demonstration project were Illinois, Kentucky, 
Oklahoma, Nevada, and South Carolina. Between 2002 and 2004, 
for each of those states the project assembled information on: 

★ the National Adult Literacy Survey to determine the literacy levels 
of the population, 

★ graduate-admission and licensure tests to establish 
the performance of the college educated, and 

★ general intellectual skills tests (WorkKeys at the two-year colleges 
and the Collegiate Learning Assessment at the four-year colleges) 
administered to a sample of students at public and private 

two- and four-year institutions. 

The learning profiles that resulted gave an idea of each state's 
strengths and challenges it faces with regard to collegiate learning. 1 
See page 3 for two examples of states' learning profiles. 2 



1 Since the sample sizes were small, it's important to look at the overall pattern 
of results rather than the individual measures. 

2 Learning profiles for Illinois, Nevada, and South Carolina, as well as Kentucky 
and Oklahoma, can be found in the full report, available at 

http: // www.highereducation.org/reports/ muleaming/index.shtml. 
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KENTUCKY 



OKLAHOMA 



Kentucky's recent substantial investments in both K-12 
and postsecondary education have been a good public- 
policy response to its low literacy levels. 

Its investments in community and technical 
colleges have paid off both in the form of higher-than- 
average proportions of graduates taking and passing 
licensing exams and in the high-level performance 
of those students on the WorkKeys exams, especially 
in the writing section. 

But the state is less competitive when it comes to 
the proportion of its graduates taking and performing 
competitively on graduate-admission exams. 

Figure 1. 

Kentucky Learning Measures 



Oklahoma's recent activity in improving the quality 
of its higher education system is a response to the 
substantial challenges it faces in its K-12 system and in 
its low levels of college graduation. The disappointing 
literacy levels of its residents reflect those challenges. 

Oklahoma's higher-education orientation toward 
workforce preparation is seen in the high number 
of students who take and do well on licensure exams, 
as compared to students' below-average performance 
on graduate admissions tests. 

Written communication skills constitute 
a particular challenge for the state in both its two- 
and four-year colleges. 

Figure 2. 

Oklahoma Learning Measures 
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Note: Learning profiles for Illinois, Nevada, and South Carolina are available at: http://www.highereducation.org/reports/muJeaming/index.shtml. 

Source for Figures and Tables: Margaret A. Miller and Peter T. Ewell, Measuring Up on College-Level Learning (San Jose, CA: The National Center for Public Policy and Higher Education, 2005). 



FURTHER ANALYZING THE RESULTS 

In addition to the general state learning profile that emerges from the data, this model reveals patterns of performance 
that suggest other state policy challenges. Two such challenges are: 

★ performance gaps by race and ethnicity, and 

★ considerable variation in how well teachers are prepared across the states. 

Information on how well it is meeting these two key national challenges is invaluable to a state, because improving 
performance in these areas is crucial to its future well-being. So what do the data tell us about how the demonstration 
states are meeting these challenges? 
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Performance Gaps by Race/Ethnicity 

As figures 3 and 4 show, white students perform better than students of color on virtually all direct learning measures. 
If these performance gaps could be eliminated or even narrowed, any state would be substantially better off. 



Figure 3. 

Work Keys Examinations: Mean Scores 

(Demonstration States: Two-Year College Students) 
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Note: To allow comparisons, the test results have been standardized by converting them 
to a scale from 0 to 1 , with 1 being the highest score possible on the test (for example, 
a score of 4 with a maximum of 7 yields a standardized score of 0.571). 

While performance gaps occur in all states, 
variability in the size of those gaps across states 
suggests that some states do a better job than others 
of meeting that challenge and may have some best 
practices to share. 

Figures 5 and 6 highlight the performance gaps 
in the demonstration states. 3 The impact of these gaps 
on overall state performance varies according to 
the size of the states' minority populations. 

★ South Carolina, for instance, takes a double hit: 
the first from the relatively large size of its 
minority population and the second from the size 
of the performance gaps. 

★ Illinois, with an even larger gap on the CLA 
problem-solving exam, still performs strongly 
on that measure because of the limited number 
of students of color in that state. 

The analysis strongly suggests that students 
of color, the fastest-growing part of our student 
population, continue to be underserved by our 
educational system even in college. Given that 
other nations are overtaking the United States 
in the proportion of young residents earning 
a baccalaureate degree, and that states' and our 
nation's future prosperity depends on these 
students, the performance gap is worthy 
of concentrated policy attention. 



3 For the results of additional WorkKeys and CLA exams, see 
http://www.highereducation.org/reports/mu_learning/index.shtml. 



Figure 4. 

Collegiate Learning Assessment: Mean Scores 

(Demonstration States: Four-Year College/University Students) 
i.o 
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Note: To allow comparisons, the test results have been standardized by converting 
them to a scale from 0 to 1 , with 1 being the highest score possible on the test. 



Figure 5. 

WorkKeys, Applied Math: Mean Scores 
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Note: To allow comparisons, the test results have been standardized by converting them 



to a scale from 0 to 1 , with 1 being the highest score possible on the test. 



Figure 6. 

CLA, Problem Solving: Mean Scores 
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* Data for Nevada were unavailable due to insufficient numbers of test-takers and 



logistical problems with test administration. 

Note: To allow comparisons, the test results have been standardized by converting them 
to a scale from 0 to 1, with 1 being the highest score possible on the test. 
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Educating Future Teachers 

Title II of the 1998 Higher Education Act requires states 
to report their pass rates on teacher licensure exams, 
which assess how much the prospective teachers know 
about the subjects they are preparing to teach. Since 
the majority of states give one of only two exams, 
states should be able to compare their performance 
on this crucial measure. 

Not so. Every state sets its own passing score 
on whatever exam it administers. The same 
performance might earn a student a passing mark 
in one state but not pass muster in another. 

The project addressed this problem by comparing 
the raw scores of students in three states (Kentucky, 
Nevada, and South Carolina) that administer many 
of the same Praxis exams. By obtaining actual test 
results from these states, the project could compare 
the students' performances by then determining what 
pass rates each state would have seen if its students 
had been held to the highest state standard for passing 
that particular exam. 

The results were startling (see Table 1). 

Kentucky's students, who had an 81.3% pass rate 
on the Praxis subject-matter exams, would have had 
only a 57.4% pass rate if they had been measured 
by the highest standard in the nation. In contrast, 

77.4% of South Carolina's students passed the exam, 
even though they did better — at 59.1% — than 
Kentucky's if measured by the highest standard. 

The situation was even more extreme in the basic- 
skills test. Praxis I (see Table 2). All Kentucky's future 
teachers passed that test (since they needed to do so 
to enter the state's teacher-education programs). 

But they performed less well than Nevada's students. 



Table 1. 


Pass Rates for Praxis II: 


Subject Knowledge 


Percentage of Students Passing 


Reported Pass Rate 


Recalculation of Pass Rate 


Based on State's 


Based on Highest State 


Own Standards 


Standard for Passing 


Kentucky 81 .3% 


57.4% 


South Carolina 77.4% 


59.1% 


Nevada 89.9% 


71 .9% 


Table 2. 


Praxis 1: Basic Skills Percentage 


of Students Passing 


Reported Pass Rate 


Recalculation of Pass Rate 


Based on State's 


Based on Highest State 


Own Standards 


Standard for Passing 


Kentucky 1 00.0% 


31.1% 


South Carolina 69.1 % 


37.8% 


Nevada 82.8% 


52.7% 



only 82.8% of whom passed the exam. If measured 
by the nation's highest standard, only 31.1% 
of Kentucky's students would have passed, 
whereas 52.7% of Nevada's would have succeeded. 

A state that is complacent about its pass rates on teacher 
licensure exams would do well to examine the numbers more 
closely, since they may mask serious underperformance 
in an area critical to the state's future. 



ABOUT THE PROJECT 

The National Forum on College-Level Learning was established in 2002 to measure what college-educated 
people know and can do in a comparable way across states. Between 2002 and 2004, the forum worked with the 
states of Illinois, Kentucky, Oklahoma, Nevada, and South Carolina on a project sponsored by The Pew Charitable 
Trusts to assess student learning. The project used national assessments of adult literacy, tests that many students 
already take when they leave college, and specially administered tests of general intellectual skills. The results made 
it possible to begin assessing both the educational capital available to the states and the contributions their colleges 
and universities collectively make to it. For project findings, see the full report. Measuring Up on College-Level 
Learning, at: http:/ / www.highereducation.org/reports/mu_learning/index.shtml. 
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WHAT'S A STATE TO DO? 



States that would like to have similar information to guide their higher education policy can replicate the project's 
methods, which are described in detail in the full report. Measuring Up on College-Level Learning (available at 
http:/ / www.highereducation.org/reports/mu_learning/index.shtml). Once they have assembled the data, 
policymakers should: 

★ convene policy and higher education leaders to discuss the findings, identify challenges, and target state policies 
and resources that could help address them; and 

★ encourage institutions to use their results to benchmark their performance and enrich their campus-based 
assessment programs. 

Margaret A. Miller, project director of the National Forum on College-Level Learning, is director of the Center 
for the Study of Higher Education at the Curry School of Education at the University of Virginia and editor 
of Change magazine. 



THE NATIONAL CENTER FOR PUBLIC POLICY AND HIGHER EDUCATION 



The National Center for Public Policy and Higher 
Education promotes public policies that enhance 
Americans' opportunities to pursue and achieve 
high-quality higher education. Established in 1998, 
the National Center is an independent, nonprofit, 
nonpartisan organization. It is not associated with any 
institution of higher education, with any political party, 
or with any government agency. 
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